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The New York City Board of Education is focused on providing a performance-driven 
educational system in which standards for student learning are clearly defined, assessment practices 
are aligned with standards, and there is continual evaluation and revision of educational strategies 
based on assessment outcomes. An important part of this effort is the implementation of a school 
quality review system, called Performance Assessment in Schools Systemwide: Essential Elements 
of Exemplary Schools (PASS). 

The PASS process provides a set of standards of exemplary educational practice and a model 
for assessing schools in relation to those standards. The process is designed to help school teams 
plan school improvement and, thereby, raise student achievement levels. PASS is grounded in the 
belief that school improvement plans must be school-based and emerge from a review conducted 
by a broad representation of the school community. 

PASS, as its name implies, possesses many of the same characteristics as a student 
performance assessment. It is rooted in standards and emphasizes instruction. It is an open, 
holistic, process-oriented assessment. PASS encourages a sense of ownership and self-reflection 
that can be invaluable in enabling school staff to determine for themselves the quality of their 
instructional program. In addition, as with all performance assessments, there are challenges to the 
uniform use of PASS 's scoring rubrics. 

i 

The following study highlights PASS 's development over the last three years, delineates the 
current model, and then describes variations in its implementation. After presenting this 
background information, the study examines threats to the reliability of the ratings which arise when 
PASS is used for individual personnel appraisals or as a result of the dual role that visiting reviewers 
play as change agents and external inspectors. 

The PASS review process is a work in progress. Over the past three years, the process has 
been collaboratively developed, field-tested, and implemented with constant feedback from the 
field. In this context, the following study represents a progress report, focusing on changes in rating 
procedures as mid-course corrections. 

DEVELOPMENT 

Research and development for PASS has largely been the responsibility of the central 
board’s Division of Assessment and Accountability (DAA) which has developed and revised the 
PASS instruments, provided training to staff from the city’s 40 local school districts, and conducted 
formative evaluations of the process. In assuming the lead on conducting actual reviews, the central 
board’s Office of Monitoring and School Improvement (OMSl) has worked in tandem with DAA to 
test the field procedures and to help refine the process. In addition, OMSl has had an external 
review function, which will be discussed later. 

Introduction of the Instrument, 1995-96 

The Deputy Chancellor for Instruction initiated the development of a school quality review 
instrument in December 1 995 in an effort to support focused planning, to improve teaching and 
learning, and to promote the high standards of achievement associated with effective schools. 
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DAA, after reviewing the literature on school reform, designed the PASS Review Guide. The 
resulting document was a product of a collaborative process utilizing input fi-om several 
constituencies with divergent backgrounds and perspectives including parents, administrators, 
supervisors, teachers. State Education Department representatives, and public advocacy groups, as 
well as central board and district staff. Schools used early versions of the document as templates 
for developing redesign plans. The language of the rubrics served as a vocabulary that described the 
kinds of exemplary practices planners wished to incorporate into their school redesigns. During 
summer 1996, the instrument was field-tested in a sample of schools. On the basis of the results the 
Review Guide was shortened. 

Piloting the Process, 1996-97 

During the 1996-97 school year, the PASS Review Guide was used in 35 New York City 
schools. OMSI and DAA conducted the reviews in low-performing schools and schools with 
special programs. These central board staff determined the extent to which schools met the PASS 
standards and modeled the PASS process for school leadership teams. Based on reviewers’ 
experiences, DAA, in consultation with OMSI, further refined the PASS Review Guide. In addition, 
DAA developed the PASS Toolkit, which provides reviewers with an overview of the steps involved 
in the review process and a comprehensive set of tools for use in its implementation. 

Systemwide Implementation, 1997-98 

In fall 1 997, the Deputy Chancellor for Instruction established policy to encourage local 
school district office staff to work directly with low-performing schools. For the first time, the 
districts were given responsibility for conducting PASS reviews jointly with school staff The 
central board provided funding to the districts for the following specific purposes: 

• to enable districts to determine how well schools were performing in comparison to a 
common set of standards; 

• to support these schools in developing better school improvement plans based on the 
PASS review findings; and 

• to support professional development activities in response to the PASS review 
findings. 

In January 1998, trained district staff began conducting PASS reviews in low-performing 
schools. Twenty-six districts conducted formal PASS reviews in 202 of New York City’s 1 145 
schools. Included were 17 elementary / middle school districts, six high school districts, and the 
district serving severely disabled students. 

The guidelines to districts for conducting PASS reviews entailed a process that differed 
significantly from OMSI reviews in that the results were not to be made available to the central 
board. District staff were asked to focus on building trust between themselves and school teams in 
order to encourage a process of self-reflection within the school communities. Districts reinforced 
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this process by allowing PASS Review Guides with completed ratings to be treated as internal 
documents, remaining within the district. 

THE PASS MODEL 

The major purpose of the PASS system is to enable school leaders to develop, review, and 
revise school plans that focus on the essential elements of effective teaching and learning. They 
gain insight into how their schools perform in relation to what is expected of an exemplary school 
by rating themselves against the performance standards and expectations that comprise the PASS 
Review Guide. Essentially, a PASS review is a collaborative and self-reflective process in which 
team members work in pairs to collect and discuss evidence and then engage in an expansive, self- 
reflective discussion to consider the evidence in light of PASS^ standards of practice. 

The review team visits classrooms and other areas of the school in pairs. In a joint review, 
each pair is comprised of a visitor and a member of the school leadership team. Trained visitors 
model the process for their partners. Through this participation, school team members gain first- 
hand experience in collecting evidence and acquire skills for self-reflection which serve them in 
creating meaningful school plans. In addition, the presence of leadership team members helps to 
put school staff at ease during classroom visits. Further, school team members provide information 
about the school and its history to the visitors, as the pairs walk around the building. 

The school is reviewed holistically. Team partners typically spend one-and-a-half days 
collecting an array of evidence throughout the school and determining as a pair the extent to which 
their evidence matches the PASS Review Guide descriptors. All review pairs reconvene for an exit 
conference where their evidence is pooled, and the whole team determines by consensus a rating 
and comments for each standard. In examining evidence, reviewers reflect on whether a pattern of 
activity or behavior supports excellence and high standards. The product is a completed PASS 
Guide which remains in the school for follow-up and planning. 

In the spirit of collaboration, visitors encourage a collegial constructive interchange of ideas 
with school leaders, so that the school personnel view the process as supportive, rather than 
intrusive or imposing. By experiencing the review with the added perspective of a critical friend, 
the leadership team undertakes more honest self-reflection and self-assessment. The visiting 
reviewers do not assume a monitoring role, but rather engage the leadership team and other school 
personnel as partners in collecting data for a joint study. As investigators rather than objects of this 
inquiry, leadership team members are more likely to use their PASS experience to revise their school 
improvement plans with a focus on the essential elements of exemplary schools. 

The school leadership team extends the knowledge and experience gained from the PASS 
process to the entire school community. Team members encourage staff, students, and parents to 
develop strategies for ongoing self-assessment based on the standards of practice in the PASS 
Review Guide. In addition, the team reviews and revises the school’s Comprehensive Education 
Plan and other plans on the basis of the PASS ratings and comments. 
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The PASS Guide is organized into ten essential elements of exemplary schools. The ten 
elements are in turn subdivided into 22 areas and withih each area is a set of indicators for 
evaluating school performance. In all, there are over 80 such indicators. The Table of Contents lists 
the 22 areas. 



School Mission/Philosophy 
Belief in Students 

School Climate 

Welcoming and Orderly Atmosphere 
Environment 

School Organization 

Organizational Plan, Structure, and Practice 
Staff Qualifications 

Curriculum and Instruction 

Alignment of Instructional Programs 
Development and Use of Instructional Materials 
Instructional Practice 

Multiple Instructional Strategies/Approaches 

Professional Development 

Development of Staff 
Formal Activities 



Instructional Resources 

Library 

Technology 

Equipment, Supplies and Materials 

Parent Involvement 

Parent Participation and Community Outreach 
Parent Education 

Support Services 
Attendance and Pupil Personnel 

External and Physical Resources 
Securing and Allocating Instructional Resources 
Non-Instructional Resources 

Assessment 

Alignment of Assessment with Curriculum 
Student Performance 
School EfTectiveness 



Measurement Scale 

The PASS Review Guide includes a measurement scale to determine the extent to which 
the school demonstrates exemplary practices within each indicator of effectiveness. Each rating 
is based on the evidence that is gathered during the school review visit. Descriptive statements 
are included next to each item on the rating scale, and reviewers must decide which of the three 
statements best describes the evidence they have gathered for that item. Evidence comes from 
written documentation, interviews with members of the school community, and observations 
during the school visit. In 1997-98, the school was given a rating for each item in the PASS 
Review Guide, as follows: 



Meeting standard There is sufficient evidence to indicate that the school meets the 

standard under review. 

Approaching standard The school does not meet the standard, but there is evidence to 

indicate that they are engaged in activities that will eventually enable 
them to meet it. 



Below standard 



There is little evidence that the school is approaching the particular 
standard. 
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In addition to providing statements within these three categories and including a column for 
the agreed-upon rating, the PASS Review Guide also provides a column for comments and 
explanations. The following page from the PASS Review Guide is illustrative of the format 
throughout. 



New York City Performance Assessment in Schools Systemwide (PASS), 1997-98 
Essential Elements of Exemplary Schools: Performance Review Guide 



Please select one rating for each item and include: source of evidence, comments , and explanations 



ELEMENT 

Area 



Meeting Standard 



Approaching Standard 



Below Standard 



Rating 
M.A. B 



Evidence, 

Comments, 

Explanation 



CURRICU- 
LUM AND 
INSTRUC- 
TION 

Multiple 

Instructional 

Strategies/ 

Approaches 



Teachers employ a 
variety of instruction^ 
approaches that are 
appropriate to the subject 
area, grade level, and 
diverse student learning 
styles. 

Teachers provide 
opportunities for students 
to work together in small 
settings and develop 
mutually supportive 
relationships. 



Teachers employ a variety 
of instructional approaches 
that are appropriate to 
subject area and grade 
level. 



Teachers provide 
opportunities for students to 
work together in small 
settings. 



Teachers employ limited 
instruction£d approaches that 
ate appropriate to subject 
area and grade level. 



Teachers provide 
opportunities for students to 
work in small settings. 



The school provides 
instruction to enable 
students with special 
needs to meet the same 
standards as other 
students. 

Students are educated in 
the least restrictive and 
most integrated 
environment possible 
consistently throughout 
the school. 



The school provides 
instruction to enable 
students with special needs 
to meet most of the same 
standards applied to other 
students. 



Students are educated in the 
least restrictive and most 
integrated environment 
possible mostly throughout 
the school. 



The school provides 
instruction to enable students 
with special needs to meet 
some of the same standards 
applied to other students. 

Students are educated in the 
least restrictive and most 
integrated environment 
possible selected 
grades/courses . 
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The three statements associated with a rating for each standard are generally differentiated 
by a few key, vmderlined words. These modifiers are the basis for distinguishing school 
performance. The modifier All" is often implicit; for example, the vmderlined words in the three 
statements may be: aligned (meeting standard), mostly aligned (approaching standard),and partially 
aligned . (below standard). Reviewers determine if the alignment is systematic and vmiform school- 
wide (for meeting standard), there is alignment in the majority of instances (approaching standard) 
or alignment in minimal cases (below standard). 

Determining Ratings 

The critical, culminating activity in a PASS review is an exit conference in which the pairs 
of reviewers -- representing insiders and outsiders — share and discuss evidence pertaining to each 
indicator. By comparing notes during the course of the review, pairs have already established a 
level of consensus about evidence they have gathered. In the exit conference, it is necessary to 
extend that consensus to the entire group. 

In deciding on a rating for a particular item, the review team keeps the following general 
considerations in mind: since the review is a brief visit and consists of a snap-shot of the life of the 
school, it is necessary to look for patterns of evidence. These patterns are the basis for generalizing 
about what is happening in the entire school over the course of the school-year. In examining 
evidence, reviewers reflect on whether a pattern of activity or behavior is occurring systematically 
and school-wide. 

The review team members collectively determine which of the three statements associated 
with an indicator best describes the evidence. The team then assigns a rating which corresponds to 
one of the three statements (i.e., meets standard, approaches standard, below standard). Reviewers 
focus on the key words that appear in the “Meeting Standard” column and determine whether the 
evidence for that item is systematic and school-wide to support a rating of meeting the standard. 

The rating is based on the school’s performance, on what is happening, or has happened. What will 
happen, should happen, or has begun to happen may be noted in the Comments section, but is not 
the determinant of the rating. Good intentions may reflect a well-conceived plan, but are not 
sufficient for meeting a standard. 

The Comments section allows reviewers to provide explanations and evidence to support 
ratings. Here, reviewers acknowledge exemplary practices and also identify specific practices 
which contribute to below standard ratings. This section also provides a space where school 
reviewers can furnish evidence that may have been overlooked, in order to support a higher rating. . 
The comments help the school team to understand the ratings so they can develop and revise plans 
for school improvement. 
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IMPLEMENTATION APPROACHES 



Within the framework of the general model, PASS has been implemented in a nvimber of 
ways. Most PASS reviews have been joint reviews, conducted by teams comprised of visitors 
trained in the PASS process together with members of the school leadership team. The visitors have 
been representatives from the 40 community district or high school superintendents’ offices, or 
central board offices such as OMSI. The goal of a joint PASS review is to provide a valid 
assessment of a school’s educational practices from the point of view of both the school leaders and 
outside visitors. In addition, the visiting review team members provide a training function by 
modeling the PASS process. In contrast, internal reviews have been conducted independently by a 
school review team, comprised of members of the school leadership team, including parents who 
have been trained in the PASS process. 

Within these variations, there have been instances of PASS being used for accountability 
purposes, rather than simply for school planning. OMSI has had a special responsibility in some of 
their joint reviews for conducting quality assurance studies. In addition, some superintendents and 
principals have used the PASS ratings as personnel performance appraisals. Both of these 
approaches — the former a legitimate alternative and the latter a conspicuous misuse - have tended 
to reinforce the imfortunate idea that PASS is a performance evaluation that can have pimitive 
consequences. 

OMSI’s External Reviews 

In addition to 1997-98's district office-sponsored PASS reviews, OMSI conducted a series of 
reviews in 3 1 schools under registration review (SURR schools) as part of an agreement with the 
State Education Department (SED). The schools were required to redesign because they had not 
met state student performance targets for a number of years. The purpose of OMSI’s reviews was 
to demonstrate that these corrective action schools had indeed been redesigned— that changes were 
basic and structural, not cosmetic and superficial. 

S.ED’s corrective action plan required that review teams go into schools not only to assess 
them on the exemplary school indicators but also to “work with school staff to build their capacity 
to conduct self-reviews and use the information to improve the implementation of the redesign 
plans.’’ As the review process has evolved, a tension has emerged from this dual mandate; 
reviewers have had to strike a delicate balance between externally-imposed rigor and sensitivity to 
the necessary dynamics for stimulating organizational change in schools. 

Insofar as the results of all OMSI reviews of redesign schools are reported back to the 
central board, OMSI's role in providing an external accountability system has persisted. At the 
same time, as the general model of PASS has become more interactive, OMSI has incorporated the 
model’s collaborative aspects. Thus, for the 1997-98 school year, when the PASS focus on 
collaboration was strengthened, OMSI reviews engaged school team members as active participants 

8 




9 



to a greater degree than in the previous year by encouraging school members of review teams to 
contribute evidence and join in the discussion of ratings. 

District Reviews 

Most district office reviewers reported that, through the pairing with school staff, they 
promote open, collaborative relationships with school teams and model the PASS process. While 
collaborative relationships between district and school staff were established in many districts, the 
process progressed unevenly. In schools where there had been little training of review team 
members, collaboration was unlikely to occur. Periodic debriefings between team members helped 
to some degree to improve collegiality and to decrease wariness on the part of the school team. 

However, before experiencing a review, because of a lack of familiarity with a holistic, self- 
reflective review, school participants had difficulty believing that the review would be for their 
benefit, to help them assess weaknesses and strengths, and to bring more focus to their planning. It 
was difficult to convince the staff, before they experienced the PASS process, that the reviewers 
were not coming in to monitor particular programs, and that the process would be collaborative. 

The apprehension was exacerbated by occasional instances of superintendents’ and . 
principals’ wanting to use review findings as evidence for a negative report on an individual staff 
member. These particular misuses reinforced general suspicion of ulterior motives, especially in the 
context of increasing accountability throughout the school system. More importantly, from the 
point of view of the PASS review process, it put school staff on the defensive, lindermining honest, 
self-reflection about teaching and learning. 

RELIABILITY OF RATINGS 

In spring 1 997, SED called for the central board to contract with an independent, external 
evaluator to “provide an independent perspective in the comprehensive evaluation of the [redesign] 
schools” . The external evaluator would conduct PASS reviews and produce reports which would 
provide information for reviewing and refining the PASS process. Bank Street College of Education 
won the contract. The focus of Bank Street's evaluation for the 1997-98 school year was to examine 
in detail .the validity and utility of the PASS review process, as implemented by OMSI in the SURR 
schools. 

Bank Street conducted reviews in a sample of ten schools drawn from the pool of corrective 
action SURR schools. These schools represent seven districts in three boroughs and close to 1 0,000 
students. OMSI had conducted two PASS reviews in each of the schools (with one exception) one 
in the fall and another in the winter. 

Because the Bank Street reviews were designed to provide an independent validation of 
OMSI’s PASS process, they diverged in two important aspects from OMSI’s implementation of 
PASS. First, unlike other SURR school PASS reviews, the ratings had no external consequences for 
the schools; and second, in a significant departure from the PASS model, school staff did not 
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participate in determining ratings, which Bank Street completed several days after the visit. The 
resultant ratings were intended for validating the process, not for providing a direct evaluation of a 
school’s performance. 

Perceived Validity of the Ratings. In Bank Street’s feedback reports on their overall 
impressions of the PASS process, the reviewers were overwhelmingly positive. As one team leader 
put it, “It presents an idea of what is happening in the school--it allows you to peel away the layers 
to examine the heart of the school. It is a positive way for the school to examine its own practices”. 
Another team leader highlighted “the interactive, open-ended nature of the Review Guide and the 
process” as the most important aspects of PASS. There was a general consensus that the process 
was “valuable and valid.” 

In all cases, reviewers reported that school personnel were open, forthcoming, and collegial 
with the Bank Street reviewers. In only two schools did reviewers also report a degree of 
guardedness on the part of some school members. This degree of openness may have been possible 
because the ratings had no external consequences for the schools; presumably, schools did not feel 
the same pressure to receive high ratings that they would during an OMSI visit. 

Most of the Bank Street reviewers agreed that their ratings reflected the overall quality of 
the schools they visited. One team leader would have welcomed the opportunity to give a more 
global, descriptive assessment of the school and the nature of the school leadership in addition to 
commenting on the individual items. However, this reviewer also acknowledged that this kind of 
commentary runs counter to “the need for the process to be reproducible by any group of observers.” 

Comparison of Ratings. When the actual ratings of OMSI’s second visit and Bank Street’s 
visit were compared, overall, using a three point scale {meeting standard = 2, approaching standard 
=1, and below standard = 0), the average ratings for each area were lower for Bank Street than 
OMSI (see Appendix A), with the exception of the area oi Instructional Practice which was rated 
almost identically by both groups. However, the relative ranking of the ratings by the two groups 
were highly correlated (r = .841, suggesting that the pattern of ratings was the similar for 

both OMSI and Bank Street. 

While there was internal consistency in the relative rank of the ratings for indicators within 
an area (.-is indicated in Appendix A), the overall ratings diverged. The following chart indicates 
that with few exceptions. Bank Street’s ratings were lower than OMSI’s. 
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TABLE 1 



Comparison of OMSI and Bank Street PASS Ratings, 1997-98* 



School OMSI 1®^ Visit OMSI 2*^ Visit Bank St Visit 





Total 

Score 


#OF 

M’s** 


Total 

Score 


#OF 

M’s** 


Total 

Score 


#OF 

M’s** 


A 


107 


35 


118 


43 


67 


13 


B 


87 


18 


124 


48 


85 


29 


C 


46 


3 


145 


64 


78 


12 


D 


137 


55 


141 


60 


149 


72 


E 


136 


54 


143 


63 


112 


34 


F 


46 


10 


128 


50 


71 


15 


G 


121 


44 


127 


49 


81 


15 


H 


137 


60 


135 


56 


85 


14 


I 


144 


62 


158 


76 


83 


16 


J 


144 


64 


na 


na 


152 


73 



* Where necessary ratings were adjusted, based on total of 82 indicators, for comparison purposes. 
**Number of the 8 1 items which reviewers rated as meeting the standard. See page 6 above for examples. 



A series of discussions between Bank Street and OMSI reviewers, which took place after 
Bank Street completed its reviews and ratings, illuminated the reasons for Bank Street’s lower rating 
pattern. Bank Street reviewers tended to use more stringent standards for giving meeting standard 
and approaching standard ratings. (In about a dozen instances. Bank Street reviewers actually gave 
ratings which were in-between meeting standard and approaching standard, or approaching 
standard and below standard, although these ratings were rounded off for purpose of the analysis.) 
Comments and discussion between the two sets of reviewers revealed a shared understanding of the 
schools' strengths and weaknesses, but OMSI often gave higher ratings in order to acknowledge 
progress from the previous visit. 

Further discussion about specific schools also revealed that in some cases OMSI reviewers 
had given a higher rating because they had more information about the school’s situation, either by 
building on information from previous visits, or because information was presented to them at the 
school’s exit conference. Since Bank Street reviews did not incorporate consensus-building, there 
was no chance for schools to provide follow-up evidence that might have been missed by reviewers 
during the course of the visit. Bank Street reviewers responded that they would have given a higher 
rating had they known this information. 

The discussions between Bank Street and OMSI about their field experiences highlighted the 
challenge of fulfilling the different PASS purposes. PASS is at once a taxonomy of standards of 
school practice and an agent for change. A balance must be struck between maintaining rigor and 
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encouraging school personnel to engage in self-reflection as one of the most productive means to 
engender school improvement 

RESPONSE TO FINDINGS 

Based on feedback from Bank Street and OMSI, as well as districts and schools which also 
conducted reviews during the 1997-98 school year, DAA staff incorporated the following changes 
into the 1 998-99 versions of the PASS Review Guide. The changes were meant to improve the 
validity of the indicators as standards of school performance. 

• A Glossary of Terms was added to the PASS guide so that criteria for ratings are 
more clearly spelled out. 

• The use of modifiers for the different ratings has been made more consistent. 

• Each indicator in the document was thoroughly reviewed; some were modified 
and/or added, based on Bank Street recommendations. 

The Five-Point Scale 

In addition, to improve the reliability of the ratings, the rating scheme was revised as a five- 
point scale to allow for acknowledgment of progress, while maintaining rigor in rating a school as 
meeting standard. School improvement could be reflected in a higher rating, without necessarily 
requiring the highest rating. 

Performance indicators are now displayed on a five point scale: 5 is a performance 
description for meeting standard, 3 is a descriptor for approaching the standard, and 1 describes 
school performance that is below the standard. The two additional rating points in this version of 
the Guide' describe varying degrees of approaching the standard: 4 is for school performance that 
is closer to meeting the standard, while 2 is to be used for school characteristics that are closer to 
below the standard. 

To determine the effect of expanding the scale, we examined OMSI’s ratings on the same 
schools for spring 1998 and winter 1999 visits. In Table 2, the comparisons for ten schools for 
which ratings were available are displayed. With the exception of one school in which a change in 
principal is recognized to have had a notable positive effect and a second school in which the ratings 
were slightly higher, the ratings for the rest of the schools are lower. 



*The PASS Performance Review Guide is available on the Internet at the Board of Education web site. It 
appears under Board of Education Information on its home page. The address is: www.nycenet.edu. 
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TABLE 2 



Comparison of OMSFs 1997-98 and 1998-99 PASS Ratings* 



OMSI 97-98 Visit OMSI 98-99 Visit 



School 


Total 


#of 


Total 


#of 




Score 


M’s 


Score 


M’s 


A 


127 


49 


115 


25 


B 


144 


64 


145 


56 


C 


119 


40 


126 


41 


D 


107 


35 


38 


0 


E 


151 


69 


148 


59 


F 


157 


75 


143 


47 


G 


138 


56 


120 


31 


H 


136 


54 


120 


33 


I 


46 


10 


108 


26 


J 


na 


na 


132 


45 



*Where necessary ratings were adjusted, based on the scoring system of the 1997-98 version 
of the PASS Review Guide, for comparison purposes. Since the scale was expanded from 
a three-point to a five-point scale, 1998-99 scores were adjusted by multiplying them by 3/5. 



Possible explanations for the lower ratings include: a drop in test scores, change in OMSI 
personnel, and differences in the time of year for the two reviews. However, OMSI representatives 
generally did not change, the schools showed a slight increase in test results, and both reviews were 
undertaken well into their respective school years. Eliminating these alternative explanations, the 
most likely cause is the expansion of the scale to five points and a greater acceptance among school 
staff who are partners in the consensus-reaching rating process that self-insight into the need for 
improvement is more important than high scores. 

Reinforcing the Value of Self-Insight 

To reinforce the value of self-insight over maximizing ratings, the PASS Toolkit for 
members of school leadership teams includes the following guideline. 



A Successful Exit Conference Produces Insight. 

PASS is not about how high the school scores, but what reviewers have 
found out about how well the school is doing compared to the standards 
of practice. This insight, rather than high scores, will enable the school 
leadership team to plan for school improvement. 
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This statement is included in the Toolkit because, even with the expanded scale, the current 
preoccupation with accoimtability issues necessitates repeating the assertion that PASS reviews are 
intended to be for planning and self-improvement, not for external evaluations, and least of all for 
individual personnel appraisals. Clearly, the best way to reassure school participants in a PASS 
review that the PASS Review Guide will be used as intended is for outside visitors on a joint review 
to be a critical friend, establishing a trusting, supportive relationship. 



CONCLUSIONS 

PASS is a complex process which, if undertaken as a rigorous, self-reflective school 
review, can be a powerful force for school improvement. It requires participating reviewers to 
reach consensus on ratings by honestly and systematically evaluating relevant evidence. The 
resulting self-insight and understanding is presumed to be a catalyst for school improvement. 

The ratings contribute to the process by providing a focus for carefully weighing the 
evidence and developing useful and rigorous comments. Comments are primary, since they 
provide a direct connection between review findings and subsequent planning. In the absence of 
ratings, we assume the review of evidence at the exit conference and development of comments 
would be indifferent and imprecise. In addition, without ratings, a comparison of an individual 
school’s PASS reviews over time can not easily reveal progress. 

The downside of providing ratings is that they are associated with external school 
evaluations as yet another accountability system. OMSI’s role in conducting PASS reviews in 
SURR schools reinforces this perception. Despite OMSI’s focus on helping school staff develop 
their own capacity for self-assessment, the school staff remain concerned about maximizing their 
scores. 



Expanding the rating scale appears to have countered the tendency for grade inflation. 
Nevertheless, the reliability of the scores is still threatened by a number of factors including: 
inadequate training of review team members; limited experience of review team members in 
exemplary schools; and continued apprehensions about the underlying purpose of the review 
process. 

DAA will address these issues as part of its ongoing development of the PASS system, 
which is viewed as an organic, interactive process -- involving PASS practitioners fi-om OMSI, 
the districts/superintendencies, and the schools themselves. The collaborative spirit of a PASS 
review, in which participants are encouraged to become members of a learning community, is 
present in the review and revision of the process itself 
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Appendix A 



Comparison of OMSI’s and Bank Street’s PASS Review Ratings 





P^SSITEM 


Reviewers 


Mean 


N 


Standard. 


Std. Error 












Deviation 


Mean 


Pair 1 


School mission 


OMSl 


5.3333 


9 


.7071 


.2357 






Bank ST 


4.4444 


9 


1.5092 


.5031 


Pair 2 


Welcoming Atmos 


OMSl 


4.3000 


10 


.9487 


.3000 






Bank St 


2.9000 


10 


1.7920 


.5667 


Pair 3 


Environment 


OMSl 


4.9000 


10 


1.1972 


.3786 






Bank St 


3.6000 


10 


1.4298 


.4522 


Pair 4 


Org plan 


OMSl 


9.2500 


10 


1.1365 


.3594 






Bank ST 


5.9300 


10 


2.4775 


.7834 


Pair 5 


Staff qualifications 


OMSl 


7.2000 


10 


1.2293 


.3887 






Bank ST 


5.8700 


10 


1.7739 


.5610 


Pair 6 


Instruct implement 


OMSl 


2.9000 


10 


1.1005 


.3480 






Bank ST 


1.7000 


10 


1.5670 


.4955 


Pair 7 


Instruct characterist 


OMSl 


8.4000 


10 


1.0750 


.3399 






Bank St 


5.2000 


10 


2.7809 


.8794 


Pair 8 


Instruct practice 


OMSl 


7.3000 


10 


1.8886 


.5972 






Bank St 


7.2000 


10 


2.7406 


.8667 


Pair 9 


Instruct strategy 


OMSl 


5.6000 


io 


1.3499 


.4269 






Bank St 


3.2300 


10 


2.4667 


.7800 


Pair 10 


Developmnt of staff 


OMSl 


5.9000 


10 


.3162 


.1000 






Bank St 


4.5500 


10 


1.2572 


.3976 


Pair 1 1 


Formal prof devlpmt 


OMSl 


11.6000 


10 


.5164 


.1633 






Bank St 


9.8200 


10 


1.8220 


.5762 


Pair 12 


Library 


OMSl 


3.7778 


9 


1.5635 


.5212 






Bank St 


2.1111 


9 


1.5366 


.5122 


Pair 13 


Technology 


OMSl 


5.7000 


10 


2.9078 


.9195 






Bank St 


4.4800 


10 


2.9951 


.9471 


Pair 14 


Equipment 


OMSl 


6.9000 


10 


1.2867 


.4069 






Bank St 


2.9000 


10 


1.1972 


.3786 


Pair 15 


Parent participation 


OMSl 


5.9000 


10 


.3162 


.1000 






Bank St 


2.8000 


10 


2.1499 


.6799 


Pair 16 


Parent education 


OMSl 


5.1000 


10 


1.1005 


.3480 






Bank St 


4.0000 


10 


1.8856 


.5963 


Pair 17 


Pupil personnel 


OMSl 


5.8000 


10 


.4216 


.1333 






Bank St 


5.0000 


10 


1.2472 


.3944 


Pair 18 


Supple resources 


OMSl 


5.2000 


10 


.7888 


.2494 






Bank St 


3.6000 


10 


2.0656 


.6532 


Pair 19 


Non inst resources 


OMSl 


7.9000 


10 


.3162 


.1000 






Bank St 


5.9000 


10 


1.6289 


.5151 


Pair 20 


School assessment 


OMSl 


4.4500 


10 


1.0659 


.3371 






Bank St 


3.6000 


10 


1.3499 


.4269 


Pair 2 1 


Student performance 


OMSl 


6.2000 


10 


1.6193 


.5121 






Bank St 


4.3000 


10 


2.3823 


.7534 
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Pair 22 School effectiveness 


OMSI 


6.0000 


10 


.0000 


.0000 




Bank St 


3.1500 


10 


1.9444 


.6149 


TOTAL SCORE 


OMSI 


136.0000 


10 


12.3828 


3.9158 




Bank St 


96.3000 


10 


30.9589 


9.7901 
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